1
00:00:00,970 --> 00:00:06,640
So we're getting up to the end of this module and the last bit that we're gonna do is we're going to

2
00:00:06,640 --> 00:00:12,400
make a prediction using our tensor flow model for how it's going to evaluate its performance on the

3
00:00:12,400 --> 00:00:13,970
test dataset.

4
00:00:14,050 --> 00:00:17,640
First let's tackle how to make a single prediction.

5
00:00:18,310 --> 00:00:24,380
So when I had to mark down cells so we've got a subsections ready that first markdown cell is going

6
00:00:24,380 --> 00:00:27,670
to read make a prediction.

7
00:00:27,760 --> 00:00:34,360
And that second markdown cell is going to read testing and evaluation.

8
00:00:34,360 --> 00:00:40,540
Notice I'm inserting all of this before closing our session and closing our writers and resetting the

9
00:00:40,540 --> 00:00:43,050
graph in your downloads.

10
00:00:43,090 --> 00:00:50,920
When you downloaded the zip file it with the amnesty data I've included a single test image and that's

11
00:00:50,980 --> 00:00:52,470
this one right here.

12
00:00:52,540 --> 00:01:00,730
This test underscore image top PMG file that test image should be in your M.A. folder which is which

13
00:01:00,730 --> 00:01:03,740
is in the same directory as your Jupiter notebook.

14
00:01:03,760 --> 00:01:10,430
Scrolling up to our imports we can write something like From pile import Image.

15
00:01:10,480 --> 00:01:16,090
So this is the pillow module which is really great for manipulating and displaying images that shift

16
00:01:16,090 --> 00:01:20,400
enter on the cell and I'll come down to where I want to make my prediction.

17
00:01:21,010 --> 00:01:28,210
And now I can see image onto open and then I can supply the relative path.

18
00:01:28,270 --> 00:01:33,880
So that's amnesty forward slash test on the square image don't P PMG.

19
00:01:34,680 --> 00:01:37,810
And if I had shift enter then I see it as an output here.

20
00:01:39,270 --> 00:01:47,930
But if I have a pin file How do I converted so that I can feed it into tensor flow and get a prediction.

21
00:01:47,940 --> 00:01:49,200
How does that work.

22
00:01:49,200 --> 00:01:54,150
Well the first thing I want to do is I might actually sort of send a variable called image and if I

23
00:01:54,150 --> 00:01:56,920
still want to display it I can write it like so.

24
00:01:57,400 --> 00:02:05,340
Well right now I have a g image but at the other end I've got a tensor that will expect seven hundred

25
00:02:05,550 --> 00:02:08,050
and eighty four inputs.

26
00:02:08,160 --> 00:02:12,610
So I've got to get the grayscale image for this thing and I have to flatten it.

27
00:02:12,630 --> 00:02:17,730
The other thing that I have to do with this image is actually to invert this image because at the moment

28
00:02:18,180 --> 00:02:22,020
it's black ink on a white background.

29
00:02:22,020 --> 00:02:29,270
But I've trained my neural network on a black background with a white digit right.

30
00:02:29,280 --> 00:02:33,200
This is a little different from what I've got at the moment.

31
00:02:33,570 --> 00:02:35,800
So I have to do quite a few things already.

32
00:02:35,820 --> 00:02:41,550
The first thing I'm going to do is actually convert this thing to grayscale and I can do that using

33
00:02:41,550 --> 00:02:51,210
pillow because pillow has a convert function and with convert I can select a mode and there is a mode

34
00:02:51,210 --> 00:02:57,690
here that is called l and this is when translating a color image to black and white.

35
00:02:57,720 --> 00:03:02,400
So at the moment even though it looks like it's black and white I think I've actually got a color image

36
00:03:02,400 --> 00:03:02,760
here.

37
00:03:03,360 --> 00:03:14,250
So when I convert this to black and white with mode L so colors BW is equal to image dot convert and

38
00:03:14,250 --> 00:03:17,220
then l what's next.

39
00:03:17,220 --> 00:03:23,320
Now I've got a black and white image I can use num pi to invert this thing believe it or not.

40
00:03:23,460 --> 00:03:31,830
So I'm going to say n p dot invert open parentheses BW which is my black and white image.

41
00:03:32,340 --> 00:03:37,770
And this will be my image and a score array so far so good.

42
00:03:37,830 --> 00:03:42,860
Now that I've done that I can take a look at the shape of my image right.

43
00:03:43,080 --> 00:03:46,420
And this is 28 by 28 by.

44
00:03:46,470 --> 00:03:47,450
Well nothing here.

45
00:03:47,460 --> 00:03:47,720
Right.

46
00:03:47,730 --> 00:03:50,910
Because it's a black and white image with one color channel.

47
00:03:50,970 --> 00:03:58,800
So that third dimension has already been dropped but it's still 28 by 28 and not flattened to seven

48
00:03:58,800 --> 00:04:00,610
hundred and eighty four.

49
00:04:00,720 --> 00:04:05,810
So we can do that with another name pie method and that's called Ravel.

50
00:04:05,880 --> 00:04:14,610
I'll store the result of this in a variable called Test underscore image and so that equal to image

51
00:04:14,710 --> 00:04:15,810
and a square array.

52
00:04:15,990 --> 00:04:20,430
Dot Ravel parentheses at the end.

53
00:04:21,000 --> 00:04:23,300
Hitting shift enter on my keyboard.

54
00:04:23,370 --> 00:04:27,090
We'll show you what the description is for this method.

55
00:04:27,240 --> 00:04:32,250
Reading the docs string we see that Ravel will return a flattened array.

56
00:04:32,280 --> 00:04:39,780
We don't want a 28 by 28 array but we actually want is a flattened array and we can see this when we

57
00:04:39,780 --> 00:04:44,200
pull up test to underscore image dot shape.

58
00:04:44,220 --> 00:04:47,570
Now we're ready to feed this into tensor flow.

59
00:04:47,730 --> 00:04:54,570
To do that we're gonna have to get a hold of our session our session object was called Sex and the way

60
00:04:54,570 --> 00:04:58,280
we get our session to actually do some calculations for us.

61
00:04:58,410 --> 00:05:06,060
We're going to have to see says Dot run the run method we'll do some calculations for us and the way

62
00:05:06,060 --> 00:05:10,830
we get this run method do some calculations for us is to provide two things.

63
00:05:10,830 --> 00:05:14,820
The first thing was the kind of output that we want from it.

64
00:05:14,820 --> 00:05:19,540
In this case we want a prediction the outputs are called fetches.

65
00:05:19,560 --> 00:05:25,480
So if we supply the fetches we can specify what kind of output we want.

66
00:05:25,500 --> 00:05:32,670
The other thing that we have to supply is the feed dictionary it's the feed dictionary are the inputs

67
00:05:32,670 --> 00:05:33,680
that's the data.

68
00:05:33,780 --> 00:05:40,030
Our session is going to use to run its calculations the feed dictionary is always a python dictionary.

69
00:05:40,080 --> 00:05:46,860
So we have to supply an X and this is what's going to go into our placeholder tensor.

70
00:05:46,890 --> 00:05:53,070
In this case it's gonna be our test underscore image right.

71
00:05:53,070 --> 00:05:56,600
Our test image will go into the place holder tensor.

72
00:05:56,760 --> 00:06:02,910
I'm not going to supply a Y because I don't really care what the accuracy or the losses so I'm just

73
00:06:02,910 --> 00:06:04,450
going to supply the X..

74
00:06:04,620 --> 00:06:11,580
Now I have to specify in the fetches what kind of output I want what I want to know is from our output

75
00:06:11,580 --> 00:06:14,520
layer from all the outputs all 10 of them.

76
00:06:14,520 --> 00:06:23,220
Which one hand the highest probability so it's gonna give us 10 outputs and I don't want to know which

77
00:06:23,220 --> 00:06:25,530
one had the highest number.

78
00:06:26,040 --> 00:06:28,490
And I want to use T F arg.

79
00:06:28,500 --> 00:06:37,770
Max open parentheses output which is the output from our last layer and then I supply the Axis axis

80
00:06:38,070 --> 00:06:43,100
is equal to 1 meaning what's the largest number in that row.

81
00:06:43,110 --> 00:06:47,920
That's how we get tensor flow to make us a prediction based on some data.

82
00:06:48,300 --> 00:06:52,460
But we're actually going to get a return value for this code.

83
00:06:52,590 --> 00:06:56,430
So I'm going to save this under prediction.

84
00:06:56,430 --> 00:07:00,520
Equals since we're fetching a number from our session.

85
00:07:00,570 --> 00:07:06,030
I to store that in a variable called prediction before I run all my code though I want to print out

86
00:07:06,030 --> 00:07:06,810
my prediction.

87
00:07:06,910 --> 00:07:08,400
Someone has enough string him.

88
00:07:08,400 --> 00:07:11,750
I'll see prediction for test.

89
00:07:11,760 --> 00:07:17,270
Image is curly braces and right prediction.

90
00:07:17,340 --> 00:07:23,580
Now I can go back up him to where I'm setting up my tensor flow graph and I'm just going to run all

91
00:07:23,580 --> 00:07:24,800
the cells below.

92
00:07:24,930 --> 00:07:28,040
And here we go our model got it right.

93
00:07:28,050 --> 00:07:29,670
Hurray.

94
00:07:29,690 --> 00:07:31,590
I've got a challenge for you.

95
00:07:31,980 --> 00:07:40,390
I'd like you to calculate the accuracy of our model over the test dataset our test dataset has 10000

96
00:07:40,440 --> 00:07:47,070
images and we've got all the features stored in X underscore test and we've got all the labels stored

97
00:07:47,490 --> 00:07:49,380
in y underscore test.

98
00:07:49,380 --> 00:07:57,240
What I'd like you to do is use your knowhow of how to use the session to calculate the accuracy over

99
00:07:57,300 --> 00:07:58,650
the test data set.

100
00:07:58,680 --> 00:08:05,580
Once you've calculated the accuracy I'd like you to print it out and display the accuracy down to two

101
00:08:05,580 --> 00:08:07,290
decimal numbers.

102
00:08:07,290 --> 00:08:11,510
I'll give you a few seconds to pause the video and give this a go.

103
00:08:11,540 --> 00:08:15,290
Far show you the solution.

104
00:08:15,340 --> 00:08:16,710
Here it goes.

105
00:08:16,750 --> 00:08:27,420
So test underscore score accuracy is equal to says Dot run open parentheses fetches is equal to accuracy.

106
00:08:27,550 --> 00:08:32,660
Why can I use this because I've defined what the accuracy is up here.

107
00:08:32,800 --> 00:08:39,220
TENSOR flow already knows how to calculate the accuracy because we've nailed it down in this cell right

108
00:08:39,220 --> 00:08:40,000
here.

109
00:08:40,030 --> 00:08:46,610
That's why we can reuse this calculation here in the fetches the fetches are what has to be output ID

110
00:08:47,080 --> 00:08:50,050
by intercession when it runs.

111
00:08:50,050 --> 00:08:57,040
The only thing it needs to know is what to give to the place holders what data to feed in and that happens

112
00:08:57,340 --> 00:09:05,110
in the feed dictionary the feed dictionary is going to have an X this time for the features and those

113
00:09:05,110 --> 00:09:09,670
are gonna be equal to while x on the score test.

114
00:09:09,730 --> 00:09:19,030
That's where our test data set has the features and the Y is going to be mapped to y underscore test

115
00:09:19,270 --> 00:09:27,130
our place holder for the labels it's gonna get the testing labels with y on a score test to print the

116
00:09:27,130 --> 00:09:35,920
test accuracy out to two decimal places when using f string and we'll see accuracy on tests it is curly

117
00:09:35,920 --> 00:09:43,120
braces test on the score accuracy and then I'm gonna use the colon to format the thing and say zero

118
00:09:43,120 --> 00:09:45,520
point two percent.

119
00:09:45,730 --> 00:09:51,580
I can't actually just hit shift enter to execute the cell because I've already closed my session.

120
00:09:51,850 --> 00:09:53,330
I'm going to have to come back.

121
00:09:53,590 --> 00:09:56,480
I'm just gonna rerun all my cells below.

122
00:09:56,500 --> 00:09:59,680
So what kind of accuracy would be a good result.

123
00:09:59,680 --> 00:10:07,420
Well I would expect our model to give us a accuracy on our test data set that is somewhat close maybe

124
00:10:07,420 --> 00:10:10,240
slightly lower than what I see on the validation data set.

125
00:10:10,660 --> 00:10:18,700
So I'm hoping that I'm getting something on the 98 97 percent mark maybe 96 if I'm not lucky.

126
00:10:18,700 --> 00:10:25,330
If you take a look around online and take a look at what the baseline results are for accuracy on the

127
00:10:25,330 --> 00:10:31,930
end list database then you'll find that a lot of models actually are able to achieve a prediction era

128
00:10:32,380 --> 00:10:34,390
of less than 1 percent.

129
00:10:34,570 --> 00:10:40,300
And if you're talking really state of the art prediction errors then we're talking around zero point

130
00:10:40,330 --> 00:10:48,070
two percent and those kind of prediction errors are achieved by very large convolution all neural networks

131
00:10:48,430 --> 00:10:49,190
in our case.

132
00:10:49,240 --> 00:10:55,840
We've actually got a very very simple precept Tron and we're training it on what really his consumer

133
00:10:55,840 --> 00:11:01,850
hardware which is not really all that expensive and not all that powerful.

134
00:11:01,930 --> 00:11:09,710
So getting a 97 98 percent test accuracy is actually really really cool.

135
00:11:09,730 --> 00:11:16,570
One of the reasons why this dataset and this problem is actually so famous is because recognizing handwritten

136
00:11:16,570 --> 00:11:23,410
digits is actually a very very practical problem and it's something that's actually used by the postal

137
00:11:23,410 --> 00:11:24,580
services.

138
00:11:24,580 --> 00:11:32,320
So for example the Postal Service in the United States processes around 500 million pieces of mail per

139
00:11:32,320 --> 00:11:40,510
day and recognizing handwritten addresses handwritten digits means that postal workers don't individually

140
00:11:40,510 --> 00:11:47,520
have to handle and examine millions and millions of mail and parcels every day corrected recognizing

141
00:11:47,530 --> 00:11:54,070
handwritten addresses is definitely a step up from processing individual digits but it's not too far

142
00:11:54,070 --> 00:11:57,670
off and we're still learning to walk before we can jog.

143
00:11:58,600 --> 00:12:03,860
If you're curious how other models have done with the atmos database I recommend you to go to this Web

144
00:12:03,870 --> 00:12:09,010
site and here you can see all the classifiers that people have trained on amnesty.

145
00:12:09,370 --> 00:12:16,680
So these researchers have collated a wonderful Web site and they are reporting the testing error in

146
00:12:16,690 --> 00:12:17,530
this table here.

147
00:12:17,770 --> 00:12:24,490
So for example boosted stumps here by these researchers in 2009 had about seven point seven percent

148
00:12:24,520 --> 00:12:25,400
test there.

149
00:12:25,870 --> 00:12:30,050
So this will give you a little bit of an idea of the lay of the land.

150
00:12:30,280 --> 00:12:36,760
And it's very interesting to see how the best non convolution of neural net fit on amnesty.

151
00:12:36,850 --> 00:12:43,080
So I've actually had a look around who built the most accurate multilayer perception that's out there

152
00:12:43,510 --> 00:12:51,910
and what I found was that in 2010 these four researchers had a perception that was massive and it achieved

153
00:12:52,090 --> 00:12:57,600
like a really really low error rate something like zero point three five percent.

154
00:12:57,760 --> 00:13:03,640
And the way they did this was to have a massive neural network that had something like six layers and

155
00:13:03,640 --> 00:13:07,320
that first hidden layer had two thousand five hundred neurons.

156
00:13:07,390 --> 00:13:10,060
The second hidden layer had two thousand neurons.

157
00:13:10,060 --> 00:13:15,700
The third one one thousand five hundred and so on all the way down to 1000 five hundred.

158
00:13:15,700 --> 00:13:21,430
And then finally ten and the other thing they did was they started messing around with the training

159
00:13:21,430 --> 00:13:21,800
data.

160
00:13:21,960 --> 00:13:29,440
He started stretching the images flipping the images and kind of like deforming the images and transforming

161
00:13:29,440 --> 00:13:35,630
them so that they were actually generating more training data than the original sixty thousand.

162
00:13:35,630 --> 00:13:41,890
So through this process of having basically an enormous percent drawn and also generating more data

163
00:13:41,950 --> 00:13:48,580
from the existing data they were able to get the error rate down to zero point three five percent without

164
00:13:48,640 --> 00:13:52,480
using a different architecture from what we were using.

165
00:13:52,810 --> 00:13:57,390
So again congratulations for making it all the way to the end.

166
00:13:57,400 --> 00:13:59,890
This was a very very difficult module.

167
00:13:59,890 --> 00:14:02,480
There was a lot of things that we covered him.

168
00:14:02,650 --> 00:14:10,210
It was heavy on the tensor flow API and it meant we really had to get stuck in with how tensor flow

169
00:14:10,240 --> 00:14:12,050
works under the hood.

170
00:14:12,070 --> 00:14:18,550
So hats off again to the team at Karas who makes tensor flow so easy to learn and so accessible because

171
00:14:18,550 --> 00:14:22,120
a lot of things are definitely not that straightforward.

172
00:14:22,120 --> 00:14:29,830
When you first learn them but when you finally do you unlock a lot of power more power than what carries

173
00:14:29,830 --> 00:14:31,330
is actually able to give you.

174
00:14:31,510 --> 00:14:37,540
And combined with tensor board you're able to really track how your training is going how to compare

175
00:14:37,540 --> 00:14:38,920
like the different models.

176
00:14:38,950 --> 00:14:43,570
If your model is learning the way you expect it to or if there are some kind of problems do you have

177
00:14:43,570 --> 00:14:51,400
to fix tensor board is an incredible tool for working with complex models and giving you insights into

178
00:14:51,400 --> 00:14:56,320
how to improve them and how to compare them and to check how your training is going.

179
00:14:56,320 --> 00:14:58,060
So I hope you enjoyed this module.

180
00:14:58,450 --> 00:14:59,300
That's all for me.

181
00:14:59,410 --> 00:15:02,420
Well done again on all the hard work.

182
00:15:02,470 --> 00:15:03,010
Take care.